Papers with contextual language models
Weakly-supervised Deep Cognate Detection Framework for Low-Resourced Languages Using Morphological Knowledge of Closely-Related Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to cognate detection focus on orthographic, phonetic or contextual models, which under-perform for most under-resourced languages. |
| Approach: | They propose a language-agnostic weakly-supervised deep cognate detection framework for under-resourced languages using morphological knowledge from closely related languages. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on cognate detection datasets across languages and can be extended to a wide range of languages from any language family. |
Exploring Category Structure with Contextual Language Models and Lexical Semantic Networks (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent work on word embeddings reports low correlations with human ratings . contextual language models (CLMs) have been successful in acquiring semantic and world knowledge. |
| Approach: | They propose to use BERT to probe contextual language models for predicting typicality scores. |
| Outcome: | The proposed methods improve on previous studies on word embeddings and their ability to predict typicality scores. |
Combining Static and Contextualised Multilingual Embeddings (2022.findings-acl)
Copied to clipboard
| Challenge: | Static embeddings are less expressive than contextual language models, but can be more straightforwardly aligned across multiple languages. |
| Approach: | They extract static embeddings for 40 languages from XLM-R and validate them with cross-lingual word retrieval and then align them using VecMap. |
| Outcome: | The proposed approach improves multilingual representations by leveraging static embeddings and a pre-training code. |
Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks do not probe professional bias as pronoun resolution may be obfuscated by cross-correlations from other manifestations of gender prejudice. |
| Approach: | They propose to use a skew and stereotype metrics to quantify and analyse the gender bias present in contextual language models when tackling the WinoBias pronoun resolution task. |
| Outcome: | The proposed methods reduce skew and stereotype relative to the unaugmented fine-tuned BERT model. |
ConRPG: Paraphrase Generation using Contexts as Regularizer (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for paraphrase generation lack reliable supervision signals. |
| Approach: | They propose an unsupervised paradigm for paraphrase generation based on contextual language models, candidate filtering and paraphrase model training based upon the selected candidates. |
| Outcome: | The proposed paradigm outperforms existing paraphrase generation methods in supervised and unsupervised setups. |
Building Static Embeddings from Contextual Ones: Is It Useful for Building Distributional Thesauri? (2022.lrec-1)
Copied to clipboard
| Challenge: | contextual language models are dominant in the field of Natural Language Processing, but they are not suitable for all uses. |
| Approach: | They propose a method for building word or type-level embeddings from contextual models . they evaluate a large set of English nouns from the perspective of extracting semantic similarity relations . |
| Outcome: | The proposed method can be used to build word or type embeddings from contextual models . it can be exploited for a wide set of English nouns, showing it can improve distributional thesauri . |
Mind Your Bias: A Critical Review of Bias Detection Methods for Contextual Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for detection of biases in contextual language models are inconsistent and inconclusive. |
| Approach: | They propose to use word embedding association test to detect biases in contextual language models to compare them with other methods. |
| Outcome: | The proposed methods are inconsistent and inconclusive for language models with word embeddings. |
Is anisotropy really the cause of BERT embeddings not being semantic? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to train contextual language models for NLP use a lightweight approach called bi-encoder, which takes two sentences as input, but does not perform well with vanilla pre-trained Transformers. |
| Approach: | They conduct a set of experiments to improve our understanding of the lack of semantic isometry in contextualized word representations in BERT. |
| Outcome: | The proposed approach does not perform well with vanilla pre-trained Transformers. |
Unleashing the Power of Neural Discourse Parsers - A Context and Structure Aware Approach Using Large Scale Pretraining (2020.coling-main)
Copied to clipboard
| Challenge: | Discourse parsing is an important upstream task within the area of Natural Language Processing (NLP) . |
| Approach: | They propose a discourse parser that incorporates recent contextual language models to improve the performance of RST-based discourse parses. |
| Outcome: | The proposed parser outperforms existing models on two key RST datasets and on large-scale "silver-standard" discourse treebank MEGA-DT. |
Scalar Adjective Identification and Multilingual Ranking (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on scalar adjective ranking have focused on English due to the availability of datasets for evaluation. |
| Approach: | They propose a binary classification task to examine the models’ ability to distinguish scalar from relational adjectives in English. |
| Outcome: | The proposed task compares the models' ability to distinguish scalar from relational adjectives in English using monolingual and multilingual models. |
Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that contextual language models display outlier dimensions . this is true for monolingual and multilingual models, but little work has been done on multilingual contexts . |
| Approach: | They investigate outlier dimensions and their relationship to anisotropy in multilingual contexts . they focus on cross-lingual semantic similarity tasks . |
| Outcome: | The proposed model improves on cross-lingual semantic similarity tasks. |
An Unsupervised, Geometric and Syntax-aware Quantification of Polysemy (2022.emnlp-main)
Copied to clipboard
| Challenge: | Polysemy is the phenomenon where a single word form possesses two or more related senses. |
| Approach: | They propose an unsupervised framework to quantify polysemy for words in multiple languages . they use syntactic knowledge to infuse the framework with syntaktic knowledge . |
| Outcome: | The proposed framework is based on syntactic knowledge and is compared with existing methods in English, French and Spanish. |
QuASE: Question-Answer Driven Sentence Encoding (2020.acl-main)
Copied to clipboard
| Challenge: | Question-answering (QA) data often encodes essential information in many facets . a growing interest of QA has led to many large-scale QA datasets available to the community . |
| Approach: | They propose a question-answer driven sentence encoding framework to learn representations from QA data. |
| Outcome: | The proposed framework learns representations from QA data, using BERT or other state-of-the-art contextual language models. |
Language Models and Semantic Relations: A Dual Relationship (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on language models for the extraction of semantic relations have focused on injecting semantic knowledge into these models to enhance them. |
| Approach: | They propose to extract lexical semantic relations from a BERT model and inject them into it using unsupervised methods based on semantic similarity at word and sentence levels. |
| Outcome: | The proposed method allows to enrich a BERT model without using any external semantic resource. |